Graphics ORT interop design and implementation #26543

praneshgo · 2025-11-10T17:30:37Z

Description

Introduced external GPU synchronization primitives enabling interoperability with Direct3D 12 and Vulkan graphics APIs.
Added APIs to acquire, wait on, and signal external GPU synchronization fences or semaphores in execution providers.
Enabled setup of CUDA Interop Graphics (CIG) contexts to synchronize CUDA streams with graphics API command queues/devices.
Extended NVIDIA TensorRT RTX execution provider with support for importing and synchronizing external semaphores from graphics APIs.
Added execution provider interface methods for managing external synchronization primitives and interop synchronization.
Added build options to enable DirectX and Vulkan interoperability features.
Added comprehensive GPU interoperability test validating D3D12 and ONNX Runtime synchronization with TensorRT RTX.

Motivation and Context

This commit largely focuses on adding the design keeping in mind sample apps using either DX or Vulkan Graphics API. The implementation part however focuses solely on DX for now, with plans to add Vulkan implementation support soon. NV RTR RTX EP gets implementation details added and virtual base APIs are added in execution_provider.h header to have fallback path for other EPs.

- Fixed the Native Methods to include binding delegations - Cleaned up GetExtSemaphore API to take better care of return values - Made returns clearer while looping active EPs

- Fixed the pointer assignment and dereferencing in fallback path APIs - Fixed the namespace in GetOrtFenceForGraphicsInterop API - in InteropEpWait and InteropEPSignal, added return calls when stream is nullptr

- Adding a compile parameter use_dx_for_interop. When specified, it enables a macro DX_FOR_INTEROP that enables d3d12.h inclusion and all other DX specific logic for compilation - Couple of small fixes

… and EpWait APIs - Modified GetOrtFenceForGraphicsInterop API to use pointer to GraphicsInteropParams - Added documentation for exposed APIs - Updated error handling in EpSignal and EpWait APIs

…eropParams and addressing current review comments - Adding Vulkan compilation support - Adding more members to GraphicsInteropParams - Addressing current review comments

Adding CIG support for DX - expected to work with TRT 1.3. Used the corresponding local fixes in TRT to verify the working of sample app that uses CIG+NV TRT RTX EP.

- Added a struct to store extSemFence and the corresponding selectedEp at GetExtSemaphore API - Not looping for EPs in InteropWait and InteropSignal calls, using the selected EP instead.

… made an opaque struct - Bifurcated fence params from graphicsInteropParams - made SemaphoreEpMap an opaque struct, defined in ORT but app does not call it. Instead it uses a void* - Some additional code fixes called out by CodeRabbit

- Added an opaque OrtFence struct. Replaced void*/void** with OrtFence*/OrtFence** for fence related params. - Replaced passing stream handle to EP with passing stream directly. - Changed the name of SetupCigContextForEpDevice to SetupGraphicsInteropContextForEpDevice

- Made the datatypes of DX/Vulkan as void* and removed the corresponding DX/Vulkan headers in onnxruntime_c_api.h - Moved the DX/Vulkan header inclusion into execution_provider.h - Reinterpret casting into appropriate data types from void* where needed

…ort_interop

Add a to-do regarding further changes needed in fallback code Not using pVkSempahore as a pointer to VkSemaphore but the semaphore handle directly, hence renaming the variable accordlingly

…RT RTX EP device. Validated to be working as expected when interop APIs are setup and outputs garbage when interop APIs are commented out.

onnxruntime/test/providers/nv_tensorrt_rtx/nv_basic_ort_interop_test.cc

@@ -0,0 +1,438 @@
+// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.


skottmckay · 2025-12-03T01:28:23Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+typedef struct FenceInteropParams {
+  ExternalSyncPrimitive extSyncPrimitive;
+  union FencePtr {
+    void* pFence;
+    void* pVkFence;
+    void* VkSemaphore;
+  } FencePtr;
+  VulkanDeviceParams VulkanDeviceParams;
+} FenceInteropParams;
+
+typedef struct SemaphoreEpMap {
+  void* extSemFence;
+  void* selectedEp;
+} SemaphoreEpMap;


The usage of 'fence' vs 'semaphore' in a number of places is confusing/inconsistent. Would be good to explain what exactly is used for DX vs VK and why. I expected these changes were only for device to device synchronization yet based on the VK doco a vkFence is for CPU to GPU sync so it's not clear to me why we need that.

We have a SemaphoreEpMap with an extSemFence which I assume could be either but the type name doesn't indicate that.
Conflicting with that is FenceInteropParams that contains fences or a semaphore.

skottmckay · 2025-12-03T01:31:41Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+
+  /** \brief Setup Graphics Interopcontext for an execution provider device.
+   *
+   * This function enables D3D12/Vulkan interoperability by creating a CUDA context


Anything in the ORT API needs to be EP agnostic. I think it's ok for the documentation to give an example that's CUDA specific, but the general documentation on what the function does should be generic.

Updated the comments and API to be EP agnostic.

skottmckay · 2025-12-03T01:49:04Z

include/onnxruntime/core/framework/execution_provider.h

 public:
  virtual ~IExecutionProvider() = default;

+  virtual Status GetExtSemaphore(const struct GraphicsInteropParams* graphicsInteropParams, struct FenceInteropParams* fenceInteropParams, void** extSemFence) {


IExecutionProvider is an internal type. As such I wouldn't expect it's API to be using things like void** extSemFence as conversion from opaque API types to the 'real' internal types should be handled by the PluginExecutionProvider class which implements IExecutionProvider for plugin EPs.

That makes sense. I will modify the code in the next commit based on this suggestion.

skottmckay · 2025-12-03T01:52:06Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+  ORT_API2_STATUS(SetupGraphicsInteropContextForEpDevice, _In_ const OrtEpDevice* ep_device,
+                  _In_ const struct GraphicsInteropParams* graphicsInteropParams);


Naively I would have expected this to be setup on a per-Session basis. An OrtEpDevice is an Environment level type so used for all sessions.

Renamed to SetupGraphicsInteropForEpDevice to better reflect its usage. The idea is that this API must be called at the beginning in order to setup graphics interop on all the sessions associated with an OrtEpDevice correctly.

skottmckay · 2025-12-03T03:15:41Z

include/onnxruntime/core/session/onnxruntime_ep_c_api.h

+   *
+   * \since Version 1.24.
+   */
+  ORT_API2_STATUS(SetupCigContext, _In_ OrtEpFactory* this_ptr,


This functionality needs to be made generic.

Updated the API name to SetupGraphicsContext to make it more generic. The functionality is generic already, EPs are expected to have their implementation in EP code files. In case of NV TRT RTX EP, SetupGraphicsContext = SetupCigContextImpl. This assignment and implementation of SetupCigContextImpl is present in onnxruntime/core/providers/nv_tensorrt_rtx/nv_provider_factory.cc

skottmckay · 2025-12-03T03:19:09Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+   *
+   * \since Version 1.24
+   */
+  ORT_API2_STATUS(InteropEpSignal, _In_ OrtSession* session, _In_ OrtFence* ortFence, _In_ OrtSyncStream* stream, _In_ uint64_t fenceValue);


Not sure about the naming of InteropEpWait/InteropEpSignal as the usage of EP/s within a session is somewhat opaque.

I would have expected the wait/signal to be around a call to OrtSession Run, with ORT figuring out when/where an EP is connected to a fence/semaphore for input/output. Should the wait/signal instead by coupled to the Run? The current setup also means you could not have concurrent calls to Run which feels like it could be overly restrictive.

Not sure about the naming of InteropEpWait/InteropEpSignal as the usage of EP/s within a session is somewhat opaque.

InteropEpWait and InteropEpSignal are designed as generic synchronization APIs that users can invoke on demand. They operate independently of the session context. To make this explicit and remove ambiguity, I have updated both API signatures to remove the OrtSession parameter.

The current setup also means you could not have concurrent calls to Run which feels like it could be overly restrictive.

Can you kindly provide an example of this scenario, so I may better understand?

skottmckay · 2025-12-03T03:22:49Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+   *
+   * \since Version 1.24
+   */
+  ORT_API2_STATUS(GetOrtFenceForGraphicsInterop, _In_ OrtSession* session, _In_ const struct GraphicsInteropParams* graphicsInteropParams, _In_ struct FenceInteropParams* fenceInteropParams, _Outptr_ OrtFence** ortFence);


Who owns ortFence and how/when should it be released?

Comments seem to be out of date.

Why do fenceInteropParams need to be non-const?

Application owns ortFence and it is responsible for releasing it when done.
Updated comments in the latest commit.
fenceInteropParams is made const struct.

skottmckay · 2025-12-03T03:24:09Z

onnxruntime/core/providers/nv_tensorrt_rtx/nv_execution_provider.cc


+Status NvExecutionProvider::GetExtSemaphore(const struct GraphicsInteropParams* graphicsInteropParams, struct FenceInteropParams* fenceInteropParams, void** extSemFence)
+{
+  if (!info_.has_user_compute_stream) 


has_user_compute_stream is very specific to this EP. how will this work for an EP that doesn't currently support user provided streams? would they create a stream using the ORT API and pass it in somehow (e.g. in the params to one of the new API calls)? we don't really want to manually have to add an option to all the EPs for this sort of thing.

There seems to be no way today to associate an external OrtSyncStream with a session in an EP-agnostic way—something like sessionOptions.setOrtSyncStream. If there is an option to do that, we can certainly avoid passing EP-specific options like has_user_compute_stream.

…e generic - Renaming APIs to make them seem more generic - Made fenceInteropParams a const struct in GetExtSemaphore

- Modified SetupGraphicsInteropContextForEpDevice to be called SetupGraphicsInteropForEpDevice, Removed occurrences of context in all the child functions it invokes. - Removed session paramter from InteropEpSignal and InteropEpWait as it is a generic API and it is not dependent on a session

skottmckay · 2025-12-18T06:32:17Z

Would be great to get your initial thoughts on #26821. I think that's more consistent in approach to other things the plugin EP is providing, as well as with the standard Create/Release setup the ORT API uses. The proposed approach also includes memory handling which other IHVs have asked for. I think there are some gaps, but that structure feels a lot cleaner to me with more explicit usage.

I have a lot of comments about the PR having spent the last day or so going through it, but if we want to move to something closer to that setup quite a few will be invalidated.

Some apply here as well though

should the OrtSyncStream to use for an inference be passed in as part of the call to Run so that there's potentially more than one stream that can be used for different inferences/sessions
would be better to get the OrtEpDevice for an input from SessionGetEpDeviceForInputs so you know for sure that the EP will be consuming the input.
all structs need to be versioned

Are the Vulkan changes here tested? I didn't see anything Vulkan specific in the unit test.

skottmckay · 2025-12-19T05:59:57Z

Can you take a look at the PRs Nick has added with initial implementation to see if there are any gaps in what that provides?

#26828
#26829

praneshgo · 2025-12-19T14:35:05Z

Can you take a look at the PRs Nick has added with initial implementation to see if there are any gaps in what that provides?

#26828 #26829

Hi Scott
Sure. I will take a look at these PRs and provide feedback. Thanks.

skottmckay · 2025-12-22T06:00:09Z

Can you take a look at the PRs Nick has added with initial implementation to see if there are any gaps in what that provides?
#26828 #26829

Hi Scott Sure. I will take a look at these PRs and provide feedback. Thanks.

Important that the setup works for Level Zero as well, which I don't believe has an equivalent to cuCtxCreate_v4 (please correct me if I'm wrong).

Possibly we could support using external semaphores as the general-purpose usage and maybe have an 'attach to device' option to enable using cuCtxCreate_v4 as an additional optimization. If you attached to the device first, would the pattern of calling 'import semaphore' followed by wait/signal with an OrtSyncStream and the result of the import still be viable? Possibly the implementation would use different types if attached to device, but if the API is generic enough that should all be opaque to ORT.

praneshgo added 19 commits October 15, 2025 08:04

Adding a few fixes based on CodeRabbit's review comments

b65c59e

- Fixed the Native Methods to include binding delegations - Cleaned up GetExtSemaphore API to take better care of return values - Made returns clearer while looping active EPs

Few bug fixes in the fall back path and a API linking issue fix

36f6130

- Fixed the pointer assignment and dereferencing in fallback path APIs - Fixed the namespace in GetOrtFenceForGraphicsInterop API - in InteropEpWait and InteropEPSignal, added return calls when stream is nullptr

Adding additional compile parameter and macro for DX interop

db281b5

- Adding a compile parameter use_dx_for_interop. When specified, it enables a macro DX_FOR_INTEROP that enables d3d12.h inclusion and all other DX specific logic for compilation - Couple of small fixes

Added documentation for APIs, modified the error handling in EpSignal…

36086b5

… and EpWait APIs - Modified GetOrtFenceForGraphicsInterop API to use pointer to GraphicsInteropParams - Added documentation for exposed APIs - Updated error handling in EpSignal and EpWait APIs

Adding Vulkan compilation support, adding more members to GraphicsInt…

9830d42

…eropParams and addressing current review comments - Adding Vulkan compilation support - Adding more members to GraphicsInteropParams - Addressing current review comments

Adding CIG support for DX

d032260

Adding CIG support for DX - expected to work with TRT 1.3. Used the corresponding local fixes in TRT to verify the working of sample app that uses CIG+NV TRT RTX EP.

Not looping for EPs in InteropWait and InteropSignal calls

0432087

- Added a struct to store extSemFence and the corresponding selectedEp at GetExtSemaphore API - Not looping for EPs in InteropWait and InteropSignal calls, using the selected EP instead.

A few code fixes, highlighted by CodeRabbit

8931f16

Fixing windows_nvtensorrtrtx_build and build_linux_nv_only pipelines

cb90d98

Adding a small fix to further get build-linux-nv-only pipeline to pass

0861626

Moving newly added APIs to the end to avoid ABI breakage

a42b35e

Merge remote-tracking branch 'origin/main' into pgonegandla/graphics_…

ad61f8c

…ort_interop

Couple of tiny fixes

8d7a5c8

Add a to-do regarding further changes needed in fallback code Not using pVkSempahore as a pointer to VkSemaphore but the semaphore handle directly, hence renaming the variable accordlingly

Adding sample test that demonstrates DX-ORT interop feature with NV T…

613ffbf

…RT RTX EP device. Validated to be working as expected when interop APIs are setup and outputs garbage when interop APIs are commented out.

Update nv_basic_ort_interop_test.cc

9d1f961

github-advanced-security bot found potential problems Nov 10, 2025

View reviewed changes

skottmckay reviewed Dec 3, 2025

View reviewed changes

praneshgo added 2 commits December 3, 2025 09:41

[1/n] Addressing a few comments - Renaming APIs to make them seem mor…

3e30712

…e generic - Renaming APIs to make them seem more generic - Made fenceInteropParams a const struct in GetExtSemaphore

nieubank mentioned this pull request Dec 17, 2025

[Feature Request] D3D12 External Resource Interop API for Plugin EPs #26821

Open

praneshgo mentioned this pull request Dec 29, 2025

[TRTRTX EP] Implement OrtExternalResourceImporter for D3D12-CUDA interop #26829

Draft

		@@ -0,0 +1,438 @@
		// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

		ORT_API2_STATUS(SetupGraphicsInteropContextForEpDevice, _In_ const OrtEpDevice* ep_device,
		_In_ const struct GraphicsInteropParams* graphicsInteropParams);

Graphics ORT interop design and implementation #26543

Are you sure you want to change the base?

Graphics ORT interop design and implementation #26543

Uh oh!

Conversation

praneshgo commented Nov 10, 2025

Description

Motivation and Context

Uh oh!

Check warning

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

praneshgo Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skottmckay commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

skottmckay commented Dec 19, 2025

Uh oh!

praneshgo commented Dec 19, 2025

Uh oh!

skottmckay commented Dec 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

praneshgo Dec 10, 2025 •

edited

Loading

skottmckay commented Dec 18, 2025 •

edited

Loading